Papers with reward function
Stay Hungry, Stay Focused: Generating Informative and Specific Questions in Information-Seeking Conversations (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on question generation assumes knowledge of what the answer might be . instead, questioner must reason pragmatically about how to acquire new information . |
| Approach: | They propose a question generation system that generates pragmatically relevant questions in information-asymmetric conversations. |
| Outcome: | The proposed questioner significantly improves the informativeness and specificity of questions generated over a baseline model as evaluated by metrics as well as humans. |
Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog (D19-1)
Copied to clipboard
| Challenge: | Existing methods to learn dialog policy require elaborate design and user goals. |
| Approach: | They propose an algorithm that estimates the reward signal and infers the user goal in dialog sessions. |
| Outcome: | The proposed algorithm achieves higher task success than state-of-the-art models on a multi-domain task-oriented dialog dataset. |
Building Task-Oriented Visual Dialog Systems Through Alternative Optimization Between Dialog Policy and Language Generation (D19-1)
Copied to clipboard
| Challenge: | Current approaches to visual dialog learning involve an end-to-end framework that maps the multi-modal context to a deep vector and in order to decode a natural dialog response. |
| Approach: | They propose a framework that trains a RL policy for image guessing and a seq2seq model to improve dialog quality. |
| Outcome: | The proposed framework achieves state-of-the-art performance on a guessWhich task . it can be applied to a wide range of tasks including assisting blind people . |
Constructing a Japanese Rap Lyric Generation Model with GRPO (2026.acl-srw)
Copied to clipboard
| Challenge: | Rap is a vocal style rooted in Hip-Hop culture, characterized by producing rhymes in synchrony with a rhythmic beat. |
| Approach: | They propose a method for generating Japanese rap lyrics with a large language model . the model's rhyming behavior is improved by using existing Japanese rhapsodysts as training data. |
| Outcome: | The proposed method improves outputs that receive moderate or high human ratings on rhyme-related criteria. |
PEPDS: A Polite and Empathetic Persuasive Dialogue System for Charity Donation (2022.coling-1)
Copied to clipboard
| Challenge: | Empathy plays a crucial role in mediating the persuasive effects as it evokes cognitive and emotional processing conducive to persuasion. |
| Approach: | They propose to use a maximum likelihood estimate loss based model to design an efficient reward function consisting of five sub rewards viz. persuasion, emotion, Politeness-Strategy Consistency, Dialogue-Coherence and Non-repetitiveness. |
| Outcome: | The proposed system increases the rate of persuasive responses with emotion and politeness acknowledgement compared to the current state-of-the-art dialogue models while maintaining the linguistic quality. |
Improving Factual Consistency Between a Response and Persona Facts (2021.eacl-main)
Copied to clipboard
| Challenge: | Neural models for response generation produce responses that are semantically plausible but not necessarily factually consistent with persona facts. |
| Approach: | They propose to fine-tune these models by reinforcement learning and an efficient reward function that explicitly captures the consistency between a response and persona facts as well as semantic plausibility. |
| Outcome: | The proposed model improves the rate of responses that are factually consistent with persona facts over its supervised counterpart while maintaining the language quality of responses. |
Interactive Text Ranking with Bayesian Optimization: A Case Study on Community QA and Summarization (2020.tacl-1)
Copied to clipboard
| Challenge: | Existing methods that focus on learning a ranking across the whole candidate space are lacking user or task-specific training data. |
| Approach: | They propose an interactive ranking approach that actively selects pairs of candidates, from which the user selects the best. |
| Outcome: | The proposed approach outperforms existing methods in community question answering and extractive multidocument summarization and is an effective reward function for reinforcement learning. |
Semi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for dialogue policy optimization do not provide sufficient supervision signals at the end of dialogues. |
| Approach: | They propose to learn from state-action pairs of an optimal policy to provide turn-by-turn rewards. |
| Outcome: | The proposed approach outperforms competitive policy learning baselines on a benchmark multi-domain dataset. |
RL with KL penalties is better viewed as Bayesian inference (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Reinforcement learning (RL) is used in fine-tuning large language models to penalize them for undesirable features of generated sequences. |
| Approach: | They analyze challenges associated with treating a language model as an RL policy . they find that RL is equivalent to variational inference: approximating a Bayesian posterior . |
| Outcome: | The proposed approach is flawed because it turns the LM into a degenerate distribution, the authors show . they show that the proposed approach avoids the distribution collapse problem and offers a first-principles derivation for its objective. |
Mapping Language to Programs using Multiple Reward Components with Inverse Reinforcement Learning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches focus on likelihood-based training or using reinforcement learning to fine-tune models based on a single reward. |
| Approach: | They propose an approach to fine-tune programs from natural language instruction . they propose a reward function that linearly combines them and a policy for program generation . |
| Outcome: | The proposed approach achieves better performance than competing methods using Reinforcement Learning. |
Self-Correcting Code Generation Using Small Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study has demonstrated that self-correction is a powerful tool for code generation, but whether it is effective for smaller models remains unexplored. |
| Approach: | They propose a method that trains small language models to maintain correct outputs while progressively correcting incorrect outputs as turns proceed. |
| Outcome: | The proposed approach improves the ability of small language models for multi-turn code correction. |
Speaking at the Right Level: Literacy-Controlled Counterspeech Generation with RAG-RL (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches produce uniform responses, ignoring that health literacy levels affect the accessibility and effectiveness of counterspeech. |
| Approach: | They propose a Controlled-Literacy framework that generates counterspeech adapted to different health literacy levels. |
| Outcome: | The proposed framework outperforms baselines by generating more accessible counterspeech to health misinformation. |
Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent efforts to verify text accuracy provide no guarantees on their correctness . a new method to improve LLMs' verifiability is to use quotes to ground models . |
| Approach: | They propose a method that allows models to quote verbatim statements from trusted sources . they leverage a fast membership inference function to verify text against trusted corpora . |
| Outcome: | The proposed method significantly increases verbatim quotes from high-quality documents by up to 130% relative to base models while maintaining response quality. |
RiT: Rubrics-in-Thinking Reinforcement Learning for Improved Reasoning in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Reasoning Models benefit from generating intermediate reasoning steps alongside final answers. |
| Approach: | They propose a framework to introduce thinking-rubric supervision into intermediate reasoning. |
| Outcome: | The proposed framework outperforms outcome-only RL baselines on reasoning-intensive and open-ended tasks. |
Better Rewards Yield Better Summaries: Learning to Summarise Without References (D19-1)
Copied to clipboard
| Challenge: | Reinforcement Learning (RL)-based document summarisation systems produce state-of-the-art performance in terms of ROUGE scores, but high summaries receive low human judgement. |
| Approach: | They propose to learn a reward function from human ratings on 2,500 summaries to generate human-appealing summary. |
| Outcome: | The proposed reward function can generate human-appealing summaries without reference summary input. |
Evidence-Driven Retrieval Augmented Response Generation for Online Misinformation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to generate counter-misinformation responses are often trained end-to-end without external knowledge, resulting in subpar text quality and excessively repetitive responses. |
| Approach: | They propose retrieval augmented response generation for online misinformation (RARG) that collects supporting evidence and generates counter-misinformation responses via reinforcement learning from human feedback. |
| Outcome: | The proposed method outperforms baselines with extensive experiments with in- and cross-domain datasets and consistently generates high-quality counter-misinformation responses. |
Prediction Improves Simultaneous Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Current systems for simultaneous machine translation use an AGENT to control an incremental encoder-decoder model. |
| Approach: | They propose a general-purpose prediction action which predicts future words in the input stream. |
| Outcome: | The proposed agent with prediction has better translation quality and less delay compared to an agent-based system without prediction. |
SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to elicit confidence from large language models are limited to binary or inaccurate group-level confidence estimates. |
| Approach: | They propose a training framework that teaches LLMs to express more fine-grained confidence estimates. |
| Outcome: | The proposed training framework reduces the confidence calibration error and maintains the performance of the model. |
When Evolution Strategy Meets Language Models Tuning (2025.coling-main)
Copied to clipboard
| Challenge: | Autoregressive language models with pretraining often display limited capability in effectively following instructions. |
| Approach: | They propose an on-policy approach to optimize models by harnessing the principle of biological evolution, namely survival of the fittest. |
| Outcome: | The proposed method can achieve superior performance in various tasks and comparable performance in the human alignment task. |
Beyond BLEU:Training Neural Machine Translation with Semantic Similarity (P19-1)
Copied to clipboard
| Challenge: | Recent work has shown that optimizing neural machine translation systems to directly improve evaluation metrics such as BLEU can improve final translation accuracy. |
| Approach: | They propose a reward function that assigns partial credit to BLEU and provides more diversity in scores than BLUE. |
| Outcome: | The proposed reward function improves translation accuracy, semantic similarity, and human evaluation on four languages trans-lated to English and the optimization procedure converges faster. |
Learning to Compress Prompt in Natural Language Formats (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work rely on compressing long contexts into soft prompts, but soft prompt compression encounters limitations in transferability . natural language (NL) prompts are incompatible with back-propagation, and NL prompts lack flexibility in imposing length constraints. |
| Approach: | They propose a framework that compresses long prompts into NL formatted Capsule Prompts. |
| Outcome: | The proposed framework reduces 81.4% of the original length, decreases inference latency up to 4.5x, and saves 80.1% of budget overheads while providing transferability across diverse LLMs and different datasets. |
Beyond Pedagogical Principles: Multi-Horizon Preference Optimization for Efficient Socratic Tutoring (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for developing LLMs are constrained by static data or sparse reward signals in online settings. |
| Approach: | They propose a framework that iteratively refines tutor agents using a multi-horizon reward function within a dynamic teacher-student simulation environment. |
| Outcome: | The proposed framework improves model performance and balances principles and effectiveness compared to baselines. |
DORA: A Dual-Objective Reinforcement Learning Framework for Effective and Efficient Multimodal Agentic Search (2026.acl-long)
Copied to clipboard
Guangming Qin, Yuhao Deng, Yukun Zhao, Zhenyang Li, Junfeng Wang, Dawei Yin, Ye Yuan, Guoren Wang, Yizhou Yan, Chengliang Chai, Lei Cao
| Challenge: | Existing methods to train large language models overlook quality of intermediate search results . existing methods often invoke search calls during reasoning, making inference inefficient . |
| Approach: | They propose a dual-objective reinforcement learning framework to improve search strategies of MLLMs . DORA outperforms state-of-the-art methods, achieving up to 8.4% higher accuracy . |
| Outcome: | The proposed model outperforms state-of-the-art methods while reducing search calls by 9.7%. |
CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning (2022.emnlp-main)
Copied to clipboard
Zeqiu Wu, Yi Luan, Hannah Rashkin, David Reitter, Hannaneh Hajishirzi, Mari Ostendorf, Gaurav Singh Tomar
| Challenge: | Existing models for conversational question answering require specific retrievers to understand user questions. |
| Approach: | They develop a query rewriting model CONQRR that rewrites a conversational question into a standalone question. |
| Outcome: | The proposed model achieves state-of-the-art on an open-domain conversational question answering dataset and is effective for two different off-the shelf retrievers. |
FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to correct factually inaccurate outputs are lacking the semantic richness needed to properly understand its internal states of trustworthiness and honesty. |
| Approach: | They propose a framework for factuality alignment that integrates natural-language uncertainty signals with external knowledge and computes confidence scores and semantic entropy from LLM outputs. |
| Outcome: | Extensive experiments on four knowledge-intensive benchmarks show that FAITH improves the factual accuracy and truthfulness of Large Language Models (LLMs). |
PAL to Lend a Helping Hand: Towards Building an Emotion Adaptive Polite and Empathetic Counseling Conversational Agent (2023.acl-long)
Copied to clipboard
| Challenge: | The social stigma associated with mental illness prevents individuals from addressing their issues and getting assistance. |
| Approach: | They propose to build a Polite and empAthetic conversational agent PAL to lay down the counseling support to substance addicts and crime victims. |
| Outcome: | The proposed agent is scalable and can be easily modified with different modules of preference models as per need. |
Generating Labeled Data for Relation Extraction: A Meta Learning Approach with Joint GPT-2 Training (2023.findings-acl)
Copied to clipboard
| Challenge: | Relation Extraction (RE) is the task of identifying semantic relation between entities mentioned in text. |
| Approach: | They propose a framework to automatically generate labeled data for Relation Extraction . they propose 'reward function' to update pre-trained language model for RE . |
| Outcome: | The proposed framework generates labeled data for relation extraction using a pre-trained language model and a meta learning approach to improve the generated samples. |
On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization (2024.findings-emnlp)
Copied to clipboard
Yong Lin, Skyler Seto, Maartje Ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, Chen Huang, Tong Zhang
| Challenge: | Reinforcement Learning from Human Feedback (RLHF) is an effective approach for aligning language models to human preferences. |
| Approach: | They compare the accuracy of DPORM and EXRM with a reward function for scoring human preferences. |
| Outcome: | The proposed methods can approximate an EXRM on the limit infinite samples, but it is unclear how effective they are in practice. |
ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidance (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for enhancing dense retrieval with query augmentation ignore the alignment between generation and ranking objectives. |
| Approach: | They propose a unified LLM-augmented dense retrieval framework that jointly optimizes both the LLM and the retriever. |
| Outcome: | Experimental results show that ExpandR outperforms strong baselines, achieving more than 5% improvement in retrieval performance. |
One More Question is Enough, Expert Question Decomposition (EQD) Model for Domain Quantitative Reasoning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Domain-specific quantitative reasoning remains a challenge for large language models . we propose an approach to balance domain knowledge with computational efficiency . |
| Approach: | They propose an approach to balance domain knowledge with computational efficiency . it uses a two-step fine-tuning framework and a reward function to measure sub-questions' effectiveness . |
| Outcome: | The proposed approach outperforms state-of-the-art domain-tuned models and advanced prompting strategies in the financial domain. |
Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code (2024.emnlp-main)
Copied to clipboard
Hyungjoo Chae, Taeyoon Kwon, Seungjun Moon, Yongho Song, Dongjin Kang, Kai Ong, Beong-woo Kwak, Seonghyeon Bae, Seung-won Hwang, Jinyoung Yeo
| Challenge: | Large language models (LLMs) have made great progress in code generation, however, they still produce errors. |
| Approach: | They propose a RL environment that provides feedback on code editing by analyzing the performance of the revised code in unit tests. |
| Outcome: | The proposed model outperforms baselines in enhancing open-source code LLMs’ code editing, making them comparable with closed-source LLM. |
Think Smart, Not Hard: Difficulty Adaptive Reasoning for Large Audio Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to determine whether to perform reasoning lack fine-grained mechanisms to adapt reasoning length to problem complexity. |
| Approach: | They propose a difficulty-adaptive reasoning method that dynamically links reasoning length to the model’s perceived problem difficulty. |
| Outcome: | The proposed method reduces average reasoning length by 50%, achieving higher efficiency without sacrificing accuracy. |